Network based content inspection (NBCI) is a technology that accumulates data packets transmitted via a data network, reconstructs the accumulated packets into payloads of application level protocols, inspects the reconstructed payloads, and invokes predefined actions according to the result of the inspection. Network based content inspection is increasingly becoming an enabling method of monitoring network data in a number of important applications such as cyber surveillance, content access control, network traffic monitoring, anti-virus, anti-spamming, content annotation, content caching, and other applications.
One problem of past methods of NBCI is reduced network performance as a result of the time required for content reconstruction, inspection, and manipulation. Generally, network performance can become severely compromised when there are many users accessing large volumes of compressed content. As the exchange of large archived content is common over today's data networks, the inspection of such content can be highly inefficient at certain times, for example, when there is a new release of popular software, digital images, videos, ring-tones, and other compressed content that are being accessed by a large number of users within a relatively short time-frame on a network.
It is also known that certain inspection tasks, such as 100% accurate polymorphic virus scanning, are NP-Complete problems. For these tasks, and with the increase of content size, the computational resources required to complete such inspection tasks grow exponentially which translates into long network latency for NBCI systems, which in turn results in low network throughput.
Performance is not the only issue. NBCI systems have a finite number of system resources, thus, when a system is subjected to communication sessions that carry large archived payloads, system resource exhaustion will happen. As a result, the system will either stop responding to new communication sessions, or will fail to open, which means that the very function of NBCI will not be applied to the new communication sessions. Therefore, past NBCI systems are generally not stable for today's enterprise and service provider networks.
A typical enterprise or service provider may deploy NBCI systems at many network junctions. Past approaches often duplicate the inspection of different instances of the same content in each of the NBCI systems. Therefore, on the whole network level, computing resources are wasted on duplicated tasks.
In other scenarios, when many instances of the same content arrive at the same time, past NBCI system will spend system resources inspecting each of the instances. Such duplication results in more resources being required which drives up the cost of NBCI systems.
A review of the prior art indicates that several technology exist in the art that enhance the performance of NBCI systems.
For example, US 2006/0221658 (Gould) uses a programmable finite state machine implemented as an integrated circuit to improve the memory usage efficiency of applying pattern matching against data payload for the purpose of content inspection. However, as today's network payloads typically contain archived content and while pattern matching is a necessary step for several NBCI applications, significant amounts of CPU cycles and memory must still be spent on de-archiving and re-archiving the content. Moreover, this cost is encountered for the inspection of every instance of the content on every NBCI system.
U.S. Pat. No. 6,154,844 (Touboul) describes a method in which a Downloadable Security Profile (DSP) is attached to the content payload. In this system, an NBCI will not inspect the payload if the payload can be associated with a DSP. While this approach effectively reduces the computation needed for inspecting the same content in the NBCI systems along the path of the content transmission, the method of attaching a DSP to the payload will cause compatibility issues downstream as the downstream systems will have to understand this DSP. In addition, for small payloads, such as those typical for short message services (SMS), this method significantly increases the size of the resulting payload. Still further, for large, archived payloads, this method does not take advantage of the fact that some components of the payload may have already been inspected. In addition, this method does not solve the system resource exhaustion issue caused by high concurrency of network data traffic or the system resource “live-lock” issue caused by inspection of large content.
With the rapid growth of network bandwidth, from 100 Mbits, to 1 Gbits, and to 10 Gbits and beyond, the importance of NBCI performance is increasingly becoming paramount in the effective management of large, complex networks. As a result, there continues to be a need for NBCI methods that effectively and efficiently process data payloads in order to improve the efficiency, stability while reducing NBCI costs without compromising network speeds.